Ranking of de-duplication rules for software system defects

ABSTRACT

A system allows users to submit information describing system defects, for example, defects in software systems. The system stores information describing defects for multiple external systems. The system further allows users to specify de-duplication rules for determining duplicate defects submitted by users. A de-duplication rule is associated with an original system defect and specifies regular expressions that are evaluated against a target system defect. The system determines scores for ranking the de-duplication rules based on features such as the number of system defects de-duplicated using the de-duplication rules, features describing an external system associated with system defects, and so on. The score determined for a de-duplication rule is used to determine a type of interaction with a user associated with the de-duplication rule.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 USC 119(e) toU.S. Provisional Application No. 63/317,463, filed on Mar. 7, 2022,which is incorporated herein by reference in its entirety for allpurposes.

BACKGROUND Field of Art

This disclosure relates in general to managing software system defectsin general and more specifically to rule based de-duplication ofsoftware system defects.

Description of the Related Art

Software systems often have defects when the system is released, forexample, in production. The defects may be reported by users. Thedefects are stored in a repository. Developers of the software systemreview the defects in the repository and modify the source code of thesoftware system to fix the defects. The same defect may be reported bymultiple users. As a result, the repository of defects may includemultiple representations of the same defect. Different users may reportthe same defect in a different manner. As a result, it is difficult toidentify all the duplicates of a defect. Having multiple representationsof the same defect results in waste of computing resources, for example,storage resources as well as developer resources utilized for reviewing,analyzing, and attempting to fix the same defect multiple times.

SUMMARY

A system allows users to submit information describing system defects,for example, defects in software systems. The system stores informationdescribing defects for multiple external systems. The system furtherallows users to specify de-duplication rules for determining duplicatedefects submitted by users.

According to an embodiment, the system stores records describing systemdefects for a plurality of external systems. Each system defect isassociated with an external system. The system configures a userinterface for displaying information stored in records describing systemdefects. The system receives de-duplication rules submitted by users.Each de-duplication rule is configured to receive a description of aninput system defect and determine whether the input system defect is aduplicate of an original system defect associated with thede-duplication rule. The de-duplication rule specifies one or moreregular expressions. The input system defect is determined to be aduplicate of the original system defect responsive to satisfying theregular expressions.

The system executes a particular de-duplication rule (e.g., R1)configured to determine duplicate system defects for a particularoriginal system defect (e.g., D1) associated with a particular externalsystem (e.g., S1). The system determines whether a target system defect(e.g., D2) specified for the particular external system S1 satisfies theone or more regular expressions specified by the de-duplication rule R1.The system determines based on the execution of the one or more regularexpressions that the target system defect D2 specified for theparticular external system S1 is a duplicate of the original systemdefect D1. The system stores in a record describing the target systemdefect D2, information indicating that the target system defect D2 is aduplicate.

The system may close the system defect D2 once the system determinesthat defect D2 is a duplicate. Accordingly, the system defect D2 is notused for a typical workflow followed by developers for fixing a systemdefect.

According to an embodiment, the system performs ranking of rules forde-duplication of system defects. The system stores records describingsystem defects for external systems. The system further receivesde-duplication rules configured to receive a description of an inputsystem defect and determine whether the input system defect is aduplicate of an original system defect associated with thede-duplication rule. According to an embodiment, the de-duplication rulespecifies one or more regular expressions. The input system defect isdetermined to be a duplicate if it matches the one or more regularexpressions. The system determines a set of features for each of theplurality of de-duplication rules. The set of features includes afeature based on a number of system defects de-duplicated using thede-duplication rules. The system determines a score indicating asignificance of the de-duplication rule based on the set of features.The system identifies a user that submitted the de-duplication rule andperforms an interaction with the user based on the score determined forthe de-duplication rule.

According to various embodiments, the set of features considered fordetermining the score for the de-duplication rule includes featuresdescribing a particular external system for which the de-duplicationrule was specified. For example, the set of features may include one ormore of (1) a feature representing a number of system defects for theparticular external system that have been submitted, (2) a featurerepresenting a total number of users of the particular external system,(3) a feature representing a number of system features supported by theparticular external system, and (4) a feature representing a rate ofuser interactions of users with the particular external system.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a system environment for performingrule-based deduplication of defects, according to an embodiment.

FIG. 2 is a block diagram illustrating components of an online systemfor performing rule-based deduplication of defects, according to oneembodiment.

FIG. 3 is a flow chart illustrating the process of determining whether anew system defect is a duplicate of another existing system defectaccording to an embodiment.

FIG. 4 is a flow chart illustrating the process of de-duplication ofsystem defects according to an embodiment.

FIG. 5 is a flow chart illustrating the process of scoringde-duplication rules according to an embodiment.

FIG. 6 is a flow chart illustrating the process of determining errors inde-duplication rules according to an embodiment.

FIG. 7 is a block diagram illustrating a functional view of a typicalcomputer system according to one embodiment.

The figures depict various embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesof the embodiments described herein.

The figures use like reference numerals to identify like elements. Aletter after a reference numeral, such as “115 a,” indicates that thetext refers specifically to the element having that particular referencenumeral. A reference numeral in the text without a following letter,such as “115,” refers to any or all of the elements in the figuresbearing that reference numeral.

DETAILED DESCRIPTION System Environment

FIG. 1 is a block diagram of a system environment for performingrule-based deduplication of defects, according to an embodiment. Thesystem environment 100 includes an online system 120 that communicateswith users via client devices 115. The online system 120 includes adeduplication engine 160 and a system defect store 150. The onlinesystem 120 may include other components not shown in FIG. 1 , forexample, other types of data stores, and so on. The system environment100 may include other elements not shown in FIG. 1 , for example, anetwork. The online system 120 may be referred to herein as a system.Certain steps of the system may be performed by the system in an offlinefashion, for example, without using a network.

The online system 120 allows users to submit system defects in systems,for example, external systems identified by the online system.Accordingly, the online system uses crowdsourcing to identify systemdefects. The online system 120 stores system defects for multipleexternal systems.

The online system 120 may send messages to users or interact with usersbased on their submissions of system defects. For example, the onlinesystem may compensate the users for system defects submitted by them.The online system may also communicate with users to report status ofsystem defects reported by the users, for example, if a system defect isdetermined to be a duplicate of an existing system defect, the onlinesystem 120 may communicate with the user that reported the systemdefect, informing the user of the fact that the system defect wasdetermined to be a duplicate.

Different external systems may have different number of system defectsreported by users. A user providing submissions of defects orde-duplication rules is also referred to as a researcher. Typically, thenumber of system defects of an external system depends on variousfactors including a length of time that has used the online system 120for allowing users to submit system defects. Accordingly, typicallyolder systems have more defects stored in the online system 120. Thenumber of defects may also depend on the size and complexity of theexternal system. For example, larger and complex external systemstypically have more system defects stored in the online system. Thenumber of defects may also depend on the popularity of the externalsystem, for example, the number of users of the external system, therate at which users interact with the external system, and so on.External systems with a large number of defects stored in the onlinesystem 120 have a higher likelihood of duplicate system defects beingreported. Furthermore, the use of crowdsourcing increases the likelihoodof receiving duplicate system defects since users are activelyresearching to identify system defects and report them.

Receiving duplicate system defects consumes computing resources andhuman resources of the online system 120 as well as the external systemfor which the duplicate system defects were reported. For example,duplicate system defects consume storage resources. If the workflow ofthe duplicate system defect is followed, the system spends resourcesfollowing the steps for resolving the system defect. Furthermore,developer resources of organizations associated with the external systemare used. If at certain stage, a determination is made that the systemdefect is duplicate, the system defect may be closed. However, any useof computing resources up to that point in time is wasted. The onlinesystem according to various embodiments performs early detection ofduplicate system defects, thereby improving the efficiency of computingresources of the online system as well as external systems.

The online system 120 uses de-duplication rules for determining whethera system defect is a duplicate of another system defect. This allows theonline system 120 to close duplicate system defects as early as possiblein the life cycle of system defects. Furthermore, early detection ofduplicates improves the user experience of users submitting the systemdefects. For example, a researcher that finds out early about the statusof a submitted defect is likely to continue performing research andsubmit more system defects rather than a researcher who does not receiveany update from the system for a long time.

The communications of the online system 120 may depend on whether asubmission of system defect provided a duplicate system defect or a newsystem defect that was previously not available in the online system120. The online system 120 allows external users to submit submissionsof de-duplication rules. Accordingly, the online system uses crowdsourcing to obtain de-duplication rules. The online system 120 may sendmessages to users or interact with users based on their submissions ofde-duplication rules. For example, the online system may compensate theusers for de-duplication rules submitted by them.

The online system 120 interacts with client applications 125 and 135that execute on client devices. The client application 125 is a systemdefect submission application 125 that presents a user interface thatallows users to submit system defects 145 to the online system 120. Thesubmitted system defects 145 may be stored in the system defect store150.

The client application 135 is a de-duplication rule submissionapplication that that presents a user interface that allows users tosubmit de-duplication rules 155. A de-duplication rule 155 is used tocompare two system defect submissions and determine whether the twosystem defect submissions describe the same system defect. Following isan example of a de-duplication rule.

{  ″name″ : ″Weak login on www.xyz.com″ ,  ″matchSubmissionFields” : {  ″bug_url″ : { ″pattern″ ″{circumflex over ( )} ( ( http[ s]?): / /)?goldendev-cloud\\.arlo\  \.com(\/.•)?″ },   ″vrt_id” : {′″pattern″ :  ″{circumflex over ( )}broken_auth_and_session_mgnt.weak_login  \\.(http_and_https_available | https_not_available)$”;  },  ″action″ : {  ″task″ : ″change_state″ ,   ″newState″ : ″duplicate” · ,   ″comment″ :″weak login duplicate″,   “originalSubmissionRefernceNumber″ :″a384700f94″  } }

The action represents the action that the system should take if thede-duplication rule triggers, for example, as a result of match of theregular expression. For example, if the fields identified inmatchSubmissionFields are determined to match the regular expressionsspecified in matchSubmissionFields for a defect submitted, the actionspecified in the action field is performed. Accordingly, the taskspecified in the task field is performed, for example, the state of thesubmitted defect is changed to the state specified in the newState field(e.g., state changed to duplicate”). Other embodiments can use otherrepresentations of de-duplication rules based on different formats(e.g., XML), different fields or attributes as well as different valuesof attributes.

In an embodiment, the client applications 125 and 135 are webapplications that execute using a web browser. However, the clientapplications 125 and 135 can be other types of applications, forexample, applications using proprietary communication protocols tointeract with the online system 120.

The de-duplication engine 160 executes the de-duplication rules toidentify duplicate system defects. In an embodiment, the de-duplicationengine 160 compares any new system defect 145 received by the onlinesystem with system defects stored in the system defect store 150 todetermine whether the new system defect 145 received matches apreviously stored system defect in the system defect store 150. If thede-duplication engine 160 determines 165 that the new system defect 145is a duplicate of an existing system defect from the system defect store150, the de-duplication engine 160 may either reject the new systemdefect 145 or store the new system defect 145 in the system defect store150 but mark it as a duplicate. Communications performed by the onlinesystem with the users submitting the system defects depend on whetherthe system defect provided by the user is a duplicate of an existingsystem defect or an entirely new system defect.

The online system 120 and client devices 115 shown in FIG. 1 representcomputing devices. A computing device can be a conventional computersystem executing, for example, a Microsoft™ Windows™-compatibleoperating system (OS), Apple™ OS X, and/or a Linux OS. A computingdevice can also be a device such as a personal digital assistant (PDA),mobile telephone, video game system, etc.

The client devices 115 may interact with the multi-tenant system 120 viaa network (not shown in FIG. 1 ). The network uses a networking protocolsuch as the transmission control protocol/Internet protocol (TCP/IP),the user datagram protocol (UDP), internet control message protocol(ICMP), etc. The data exchanged over the network can be representedusing technologies and/or formats including the hypertext markuplanguage (HTML), the extensible markup language (XML), etc.

System Architecture

FIG. 2 is a block diagram illustrating components of an online systemfor performing rule-based deduplication of defects, according to oneembodiment. online system 120 comprises a system defect submissionmodule 210, a de-duplication rule submission module 220, thede-duplication engine 160, a rule ranking module 230, a task creationmodule 240, a system defect store 150, a de-duplication rule store 245,and a user store 250. Other embodiments can have different and/or othercomponents than the ones described here. Furthermore, thefunctionalities described herein can be distributed among the componentsin a different manner.

The system defect store 150 stores system defects submitted by users. Inan embodiment, the system defect store 150 is a database, for example, arelational database or a document-based database. The system defectsstore 150 stores various attributes describing a system defect includinginformation identifying an external system for which the system defectis submitted, information identifying a user that submitted the systemdefect, a time of submission of the system defect, and so on. In anembodiment, the system defect is specified using unstructured text, forexample, natural language description of the system defect. In anembodiment, the system defect store stores a flag indicating whether asystem defect is a duplicate of another system defect. In an embodiment,the system defect store 150 stores for a duplicate system defect, anidentifier of the earliest system defect that is duplicate of thatsystem defect. If there are multiple system defects that are duplicatesof each other, the system stores a flag identifying the earliestreported system defect as the original system defect and the remainingsystem defects as duplicates. The system defect store 150 may also storeinformation identifying a de-duplication rule used to determine that thesystem defect is duplicate of another system defect. In an embodiment,the online system closes the duplicate system defects so that only theoriginal system defect is active in the system defect store 150 and allother duplicate system defects are marked as closed or inactive.

The de-duplication rule store 245 stores de-duplication rules. In anembodiment, the system defect store 150 is a database, for example, arelational database. Each de-duplication rule has a unique ruleidentifier. The system defect store 150 may use the rule identifier as aforeign key for identifying the de-duplication rule used for determiningwhether a system defect is duplicate of another system defect. Since thede-duplication rules are specified by users via crowd sourcing,de-duplication rule store 245 stores information identifying a user thatprovided each de-duplication rule and the time when the de-duplicationrule was received by the system from the user. In an embodiment, thede-duplication rule store 245 stores statistics associated with ade-duplication rule,

The user store 250 stores information describing the users thatcontribute submissions including submissions of system defects andsubmissions of de-duplication rules. The user store 250 includes aunique identifier for each user. The unique identifier acts as a foreignkey for the records in system defect store 150 and de-duplication rulestore 245 to identify the users that provided the correspondingsubmission.

The system defect submission module 210 configures a user interface andpresents it via the system defect submission application 125 displayedvia a client device 115. The system defect submission module 210receives system defects submitted via the user interface of the systemdefect submission application 125 and provided by a user. The systemdefect submission module 210 may store the system defect in the systemdefect store 150. The system defect submission module 210 may invoke thede-duplication engine 160 to determine whether the newly submittedsystem defect is a duplicate of an existing system defect and thenstores the resulting information in the record in the system defectstore 150.

The de-duplication rule submission module 220 configures a userinterface and presents it via the de-duplication rule submissionapplication 135 displayed via a client device 115. The de-duplicationrule submission module 220 receives de-duplication rules submitted viathe user interface of the system defect submission application 125 andprovided by a user. The de-duplication rule submission module 220 maystore the system defect in the system defect store 150.

The de-duplication engine 160 executes the de-duplication rules todetermine whether a system defect is a duplicate of another systemdefect. The de-duplication engine 160 may be invoked for a newlyreceived system defect to determine whether the newly received systemdefect is a duplicate of an existing de-duplication rule. Thede-duplication engine 160 may be invoked periodically to processexisting system defects stored in the system defect store 150 todetermine whether any system defects are duplicates of other systemdefects.

The rule scoring module 230 determines a score indicating a significanceof a de-duplication rule. The rule scoring module 230 determines thescore for a de-duplication rule based on various factors including ameasure of a number of system defects that were determined to beduplicate of other system defects by applying the de-duplication rule.In some embodiments, the score of a de-duplication rule also depends onthe age of the external system corresponding to the original systemdefect of the de-duplication rule. The score of a de-duplication rule isdirectly proportional or directly related to the age of thecorresponding external system in the online system. For example, olderexternal systems are given higher score compared to recent systems. Theage of the external system represents the length of time for whichsystem defects of the external system are being submitted in the onlinesystem 120. This is so because the number of submissions of systemdefects in the online system grows over time and older external systemsin the online system 120 have more system defects and a greater need toidentify duplicate system defects. Similarly, the score of ade-duplication rule associated with an external system depends on thenumber of system defects for the external system that have beensubmitted and exist in the online system. In an embodiment, the onlinesystem receives attributes describing an external system and uses theattributes as factors for determining scores of de-duplication rules fororiginal system defects of the external system. These attributes includea measure of complexity of the external system, a number of users of theexternal system, a number of system features supported by the externalsystem (using a measure of complexity of the external system such that acomplex external system supporting more system features is ranked higherthan a less complex external system supporting fewer system features), arate of user interactions of users with the external system (indicatinga measure of how busy the external system is). For example, the score ofa deduplication rule for an original system defect associated with anexternal system is (1) directly proportional or directly related to thecomplexity of the external system, (2) directly proportional or directlyrelated to the total number of users of the external system, (3)directly proportional or directly related to the number of featuressupported by the external system, (4) directly proportional or directlyrelated to the rate of user interactions of users with the externalsystem. In an embodiment, the online system determines a score for ade-duplication rule based on a combination of different factors, forexample, a weighted aggregate of various factors described herein.

In some embodiments, the online system uses a machine learning model fordetermining the score for a de-duplication rule. The machine learningbased model receives various features identified above and encodes themas a feature vector. For example, the features may be: (1) a number ofsystem defects for the external system that have been submitted in thepast with the online system that may be stored in the online system (2)a measure of complexity of the external system, (3) the total number ofusers of the external system, (4) the number of features supported bythe external system, (5) the rate of user interactions of users with theexternal system, (6) an age of the external system indicating how longthe external system has been operational (7) a measure of age of theexternal system within the online system indicating how long systemdefects have been submitted within the online system for the externalsystem, (8) a measure of complexity of the system defects submitted forthe external system (e.g., an average measure of developer time requiredto fix system defects submitted for the external system or a measure ofan average number of lines of code modified for fixing system defects ofthe external system) and so on. A training data set is provided as inputto the machine learning model. In an embodiment, the machine learningmodel is a supervised machine learning model that is trained usinglabeled training data set. The training dataset includes de-duplicationrules and corresponding scores for the de-duplication rules that may bedetermined by experts. The system trains the machine learning basedmodel by adjusting the parameters of the machine learning based modelusing techniques such as back propagation to minimize a measure of lossbetween a predicted score for a de-duplication rule from the trainingdataset and the known score available in the training dataset. Thetrained machine learning based model is deployed in the system todetermine scores of the new de-duplication rules added to the system.

In some embodiments, the machine learning model is a neural network, forexample, a multi-layer perceptron. The input provided to the neuralnetwork is an encoding of the de-duplication rule as well as encoding ofthe information describing the rules, for example, the number of timesthe rule matches, attributes describing the external system associatedwith the de-duplication rule, and so on. The neural network is trainedto predict the score for an input de-duplication rule. The hidden layersof the neural network determine features of the de-duplication rule thatare used for determining the score. The trained neural network isdeployed in the system to determine scores of the new de-duplicationrules added to the system.

The online system performs interactions with users that submitted thede-duplication rule. The interactions are based on the score of thede-duplication rule submitted by the user. For example, the onlinesystem may provide compensation (or reward) to the user that submittedthe de-duplication rule. In an embodiment, a value of compensationprovided to the users that submitted the de-duplication rule isdetermined based on the score determined for the de-duplication rule.The compensation is determined to encourage submitters to submitde-duplication rules that are effective, instead of submitters focusingon system defects that are likely to be duplicates.

In some embodiments, the system also interacts with users that submittedsystem defects that are determined to be duplicates of existing systemdefects based on execution of the de-duplication rules. Theseinteractions may inform the user that submitted the system defect thatthe system defect submitted by the user was determined to be aduplicate. Accordingly, the system defect will not be submitted for aworkflow for fixing the system defect. Furthermore, the user is informedthat the user may not be compensated for submission of the system defectsince the system defect was determined to be duplicate of an existingdefect submitted earlier. Note that the online system compensates usersthat submitted system defects, provided the system defects are notdetermined to be duplicates of existing system defects.

The use of the scores affects the user interactions with the system andthereby improves the efficiency of the system since users are encouragedto submit more de-duplication rules and make them effective, therebyreducing the number of defects that are processed by the system. Themechanism disclosed reverses the incentives provided to users fromincentive to submit duplicative defects (in an attempt to earn rewardsprovided by the online system for submitted defects) that increases theduplicative defects in the system to an incentive to submitde-duplication rules (for earning rewards provided by the online systemfor submitted de-duplication rules) that decreases the number of(active) duplicative defects in the system. The system motivatessubmitters to submit de-duplication rules thereby automating thedetection of duplicate defects and minimizing the cost and efforts ofdealing with duplicate defects. As a result, the system consumes fewercomputing resources for processing the defects. As defects are marked asduplicates, there are fewer interactions with the system related tothese defects, thereby improving efficiency of usage of networkbandwidth of the system. Furthermore, defects that are marked asduplicate may be eliminated from the system, thereby improving memoryusage of the system as well as storage efficiency of the system.

The score of a deduplication rule is also determined based on a numberof system defects that were determined to be erroneously marked asduplicates based on a matching performed by the de-duplication rule. Thescore of a de-duplication rule may be used for ranking thede-duplication rules, for example, while displaying the de-duplicationrules via a user interface. The score of a de-duplication rule may beused to determine a type of message sent to a user that provided thede-duplication rule. The score of a de-duplication rule may be used todetermine a compensation provided to a user that provided thede-duplication rule

The task creation module 240 creates tasks associated withde-duplication rules or system defects. For example, if the system marksa system defect as duplicate and closes it and subsequently receives auser interaction associated with the system defect, the task creationmodule 240 creates a task for a system administrator to review thesystem defect and determine whether the system defect was accuratelydetermined to be a duplicate. The task may also request the systemadministrator to review the de-duplication rule that was used toerroneously determine that a system defect is a duplicate of anothersystem defect.

Processes

Various processes executed by the online system 120 are illustrated inFIGS. 3-6 . Various embodiments can perform the steps of theseflowcharts in different orders. Furthermore, various embodiments caninclude different and/or additional steps than the ones describedherein.

FIG. 3 is a flow chart illustrating the process 300 of determiningwhether a new system defect is a duplicate of another existing systemdefect according to an embodiment. The system receives 310 a descriptionof a defect in a software system provided by a user. The system mayassign a unique identifier for the received system defect. The systemexecutes 320 the de-duplication engine 160 to compare the receivedsystem defect with existing system defects. The system determines 330based on the result of execution of the de-duplication engine 160whether the received system defect is a duplicate of an existing systemdefect.

If the system determines 330 based on the result of execution of thede-duplication engine 160 that the received system defect is a duplicateof an existing system defect, the system performs the steps 340, 350,and 360. Accordingly, the system marks 340 the new system defect as aduplicate. The system may store information identifying the originalsystem defect of which the new system defect is determined to be aduplicate of. The system closes 350 the new system defect, i.e., marksthe new system defect as inactive or a system defect that does not needto be processed or reported. The system may send 360 a message to theuser that provided the system defect indicating that the new systemdefect was determined to be a duplicate of an existing system defect.

If the system determines 330 based on the result of execution of thede-duplication engine 160 that the received system defect is not aduplicate of an existing system defect, the system performs the steps370, 380, and 390. Accordingly, the system stores the new system defectas an original system defect in a repository, for example, the systemdefect store 150. The system may provide 380 information describing thenew system defect to developers of the external system for which thesystem defect was identified. The system may send 390 a message to theuser providing the system defect that the new system defect wasdetermined to be an original system defect, i.e., a system defect thatis not a duplicate of an existing system defect.

FIG. 4 is a flow chart illustrating the process 400 of de-duplication ofsystem defects according to an embodiment. The system accesses 410 ade-duplication rule. The system may access the de-duplication rulebecause the de-duplication rule was just submitted by a user.Alternatively, the system may access de-duplication rule in an offlinemode to identify duplicate system defects from previously acquiredsystem defects.

The system identifies 420 an original system defect based on thede-duplication rule. The original system defect is the system defectwith which other system defects are compared by the de-duplication rule.The system further extracts 430 one or more regular expressionsspecified by the de-duplication rule for comparing various portions ofsystem defects rules.

The system repeats the steps 440, 450, 460, and 470 for multiple targetsystem defects that were reported for the same system for which theoriginal system defect was reported. The system evaluates 440 the one ormore regular expressions for the target system defect. If the systemfinds a match based on the evaluation of the regular expressions, thesystem marks 450 the target system defect as a duplicate of the sourcesystem defect.

In an embodiment, the system compares the time stamps when each of thesource and target system defects were received by the system and marksthe system defect that was received earlier as the original systemdefect associated with the de-duplication rule and marks the systemdefect that was received later as the duplicate.

Accordingly, the system closes 460 the system defect that is determinedto be a duplicate. The system may optionally send 470 a message to theuser that provided the target system defect, for example, a messageinforming the user that the system defect provided by the user is aduplicate of another system defect.

FIG. 5 is a flow chart illustrating the process 500 of scoringde-duplication rules according to an embodiment. The system identifies510 a de-duplication rule for scoring. The system determines 520 variousfactors relevant for scoring the identified de-duplication rule based onthe significance of the de-duplication rule. For example, the systemdetermines a number of defects determined as duplicates of theidentified system defect based on one or more de-duplication rules. Thesystem determines 530 a score for the de-duplication rule based onvarious factors, for example, the number of defects determined asduplicates of the identified system defect. The system sends 540 amessage based on the score to the user that submitted the de-duplicationrule. For example, the system may report the score of the de-duplicationrule to the user. In an embodiment, the system compensates the use thatprovided the de-duplication rule based on the score of thede-duplication rule. Accordingly, the user gets higher compensation ifthe de-duplication rule provided by the user is used to detect a largenumber of duplicate system defects. The compensation of the userreflects the saving in computing resources and human resources achievedby the online system and the external system based on the de-duplicationrule. In an embodiment, the scores of de-duplication rules are used forranking the de-duplication rules. For example, the de-duplication rulesmay be displayed on a dashboard to a system administer. The onlinesystem ranks the de-duplication rules based on the scores so thatsignificant de-duplication rules with higher scores are shown higher inthe rank or are shown more prominently compared to de-duplication ruleswith lower scores.

In an embodiment, a user can submit de-duplication rules for identifyingduplicate de-duplication rules. Accordingly, the de-duplication rulecompares two de-duplication rules using regular expressions applied todifferent portions of de-duplication rules. The de-duplication ruleidentifies a source de-duplication rule and one or more regularexpressions for comparing the de-duplication rule with otherde-duplication rules. The system uses such de-duplication to comparede-duplication rules and eliminate duplicate de-duplication rules. Iftwo de-duplication rules are determined to match, the system keeps thede-duplication rule that was received earlier and removes thede-duplication rule that was provided later as the duplicate.

FIG. 6 is a flow chart illustrating the process 600 of determiningerrors in de-duplication rules according to an embodiment. Accordingly,the online system 120 catches mistakes or errors in de-duplicationrules. If the online system receives an interaction, for example, areply from a researcher for a submission that has been closed by thesystem, the online system creates a task for an expert to manuallyreview the interaction, for example, the reply or a comment by a user.If a determination is made that there is an error in the de-duplicationrule, the rule may be adjusted. If a determination is made that there isno error in the de-duplication rule, an agent or an expert may provideexplanation to the researcher describing why a duplicate state for thesystem defect is valid.

The system may close 610 a system defect based on a de-duplication rule.However, the system receives 620 a communication related to the closedsystem defect. The system sends 630 the closed system defect for review,for example, by a system administrator or an expert user. The system mayfurther send the de-duplication rule for review to an expert and confirm640 from the expert whether the de-duplication rule has an error. If thesystem determines that the de-duplication rule has error, the systemrequests modification of the de-duplication rule from the user thatprovided the de-duplication rule. The system may suspend the executionof the de-duplication rule until the error is fixed by modifying thede-duplication rule. The system may delete the de-duplication rule ifthe de-duplication rule is not modified for more than a threshold timeafter being reported as an erroneous de-duplication rule.

Computer Architecture

FIG. 7 is a high-level block diagram illustrating a functional view of atypical computer system for use as one of the entities illustrated inthe environment 100 of FIG. 1 according to an embodiment. Illustratedare at least one processor 702 coupled to a chipset 704. Also coupled tothe chipset 704 are a memory 706, a storage device 708, a keyboard 710,a graphics adapter 712, a pointing device 714, and a network adapter716. A display 718 is coupled to the graphics adapter 712. In oneembodiment, the functionality of the chipset 704 is provided by a memorycontroller hub 720 and an I/O controller hub 722. In another embodiment,the memory 706 is coupled directly to the processor 702 instead of thechipset 704.

The storage device 708 is a non-transitory computer-readable storagemedium, such as a hard drive, compact disk read-only memory (CD-ROM),DVD, or a solid-state memory device. The memory 706 holds instructionsand data used by the processor 702. The pointing device 714 may be amouse, track ball, or other type of pointing device, and is used incombination with the keyboard 710 to input data into the computer system700. The graphics adapter 712 displays images and other information onthe display 718. The network adapter 716 couples the computer system 700to a network.

As is known in the art, a computer 700 can have different and/or othercomponents than those shown in FIG. 7 . In addition, the computer 700can lack certain illustrated components. For example, a computer system700 acting as a multi-tenant system 120 may lack a keyboard 710 and apointing device 714. Moreover, the storage device 708 can be localand/or remote from the computer 700 (such as embodied within a storagearea network (SAN)).

The computer 700 is adapted to execute computer modules for providingthe functionality described herein. As used herein, the term “module”refers to computer program instruction and other logic for providing aspecified functionality. A module can be implemented in hardware,firmware, and/or software. A module can include one or more processes,and/or be provided by only part of a process. A module is typicallystored on the storage device 708, loaded into the memory 706, andexecuted by the processor 702.

The types of computer systems 700 used by the entities of FIG. 1 canvary depending upon the embodiment and the processing power used by theentity. For example, a client device 115 may be a mobile phone withlimited processing power, a small display 718, and may lack a pointingdevice 714. The multi-tenant system 120, in contrast, may comprisemultiple blade servers working together to provide the functionalitydescribed herein.

ADDITIONAL CONSIDERATIONS

The particular naming of the components, capitalization of terms, theattributes, data structures, or any other programming or structuralaspect is not mandatory or significant, and the mechanisms thatimplement the embodiments described may have different names, formats,or protocols. Further, the systems may be implemented via a combinationof hardware and software, as described, or entirely in hardwareelements. Also, the particular division of functionality between thevarious system components described herein is merely exemplary, and notmandatory; functions performed by a single system component may insteadbe performed by multiple components, and functions performed by multiplecomponents may instead performed by a single component.

Some portions of above description present features in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are the means used bythose skilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. These operations,while described functionally or logically, are understood to beimplemented by computer programs. Furthermore, it has also provenconvenient at times, to refer to these arrangements of operations asmodules or by functional names, without loss of generality.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices.

Certain embodiments described herein include process steps andinstructions described in the form of an algorithm. It should be notedthat the process steps and instructions of the embodiments could beembodied in software, firmware or hardware, and when embodied insoftware, could be downloaded to reside on and be operated fromdifferent platforms used by real time network operating systems.

The embodiments described also relate to apparatuses for performing theoperations herein. An apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored on acomputer readable medium that can be accessed by the computer. Such acomputer program may be stored in a non-transitory computer readablestorage medium, such as, but is not limited to, any type of diskincluding floppy disks, optical disks, CD-ROMs, magnetic-optical disks,read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, application specific integratedcircuits (ASICs), or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus. Furthermore,the computers referred to in the specification may include a singleprocessor or may be architectures employing multiple processor designsfor increased computing capability.

The algorithms and operations presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may also be used with programs in accordancewith the teachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will be apparent to those ofskill in the, along with equivalent variations. In addition, the presentembodiments are not described with reference to any particularprogramming language. It is appreciated that a variety of programminglanguages may be used to implement the teachings of the embodiments asdescribed herein.

The embodiments are well suited for a wide variety of computer networksystems over numerous topologies. Within this field, the configurationand management of large networks comprise storage devices and computersthat are communicatively coupled to dissimilar computers and storagedevices over a network, such as the Internet.

Finally, it should be noted that the language used in the specificationhas been principally selected for readability and instructionalpurposes, and may not have been selected to delineate or circumscribethe inventive subject matter. Accordingly, the disclosure of theembodiments is intended to be illustrative, but not limiting.

What is claimed is:
 1. A computer implemented method for ranking ofrules for de-duplication of system defects, the method comprising:storing records describing system defects for a plurality of externalsystems, each system defect associated with an external system;receiving, a plurality of de-duplication rules, wherein a de-duplicationrule is configured to receive a description of an input system defectand determine whether the input system defect is a duplicate of anoriginal system defect associated with the de-duplication rule, thede-duplication rule specifying one or more regular expressions, whereinthe input system defect is determined to be a duplicate responsive tothe input system defect matching the one or more regular expressions;for each of the plurality of de-duplication rules: determining a set offeatures comprising, a feature based on a number of system defectsde-duplicated using the de-duplication rules, and determining a scoreindicating a significance of the de-duplication rule based on the set offeatures; identifying a user that submitted the de-duplication rule; andperforming an interaction with the user associated with thede-duplication rule based on the score determined for the de-duplicationrule.
 2. The method of claim 1, wherein the set of features comprisesone or more features describing a particular external system for whichthe de-duplication rule was specified.
 3. The method of claim 2, whereinthe set of features comprises a feature representing a number of systemdefects for the particular external system that have been submitted. 4.The method of claim 2, wherein the set of features comprises a featurerepresenting a total number of users of the particular external system.5. The method of claim 2, wherein the set of features comprises afeature representing a number of system features supported by theparticular external system.
 6. The method of claim 2, wherein the set offeatures comprises a feature representing a rate of user interactions ofusers with the particular external system.
 7. The method of claim 1,wherein one or more system defects include unstructured text describinga difference between an implemented feature of the external system andan expected feature.
 8. The method of claim 1, further comprising:ranking a plurality of de-duplication rules based on the scores of eachof the plurality of de-duplication rules; and sending the rankedde-duplication rules for presentation via a user interface.
 9. Themethod of claim 1, wherein a record describing a system defect comprisesone or more components, wherein each regular expression of ade-duplication rule is associated with a component of the system defectof the de-duplication rule.
 10. A non-transitory computer readablestorage medium storing instructions that when executed by one or morecomputer processors cause the one or more computer processors to performsteps comprising: storing records describing system defects for aplurality of external systems, each system defect associated with anexternal system; receiving, a plurality of de-duplication rules, whereina de-duplication rule is configured to receive a description of an inputsystem defect and determine whether the input system defect is aduplicate of an original system defect associated with thede-duplication rule, the de-duplication rule specifying one or moreregular expressions, wherein the input system defect is determined to bea duplicate responsive to the input system defect matching the one ormore regular expressions; for each of the plurality of de-duplicationrules: determining a set of features comprising, a feature based on anumber of system defects de-duplicated using the de-duplication rules,and determining a score indicating a significance of the de-duplicationrule based on the set of features; identifying a user that submitted thede-duplication rule; and performing an interaction with the userassociated with the de-duplication rule based on the score determinedfor the de-duplication rule.
 11. The non-transitory computer readablestorage medium of claim 10, wherein the set of features comprises one ormore features describing a particular external system for which thede-duplication rule was specified.
 12. The non-transitory computerreadable storage medium of claim 11, wherein the set of featurescomprises a feature representing a number of system defects for theparticular external system that have been submitted.
 13. Thenon-transitory computer readable storage medium of claim 11, wherein theset of features comprises a feature representing a total number of usersof the particular external system.
 14. The non-transitory computerreadable storage medium of claim 11, wherein the set of featurescomprises a feature representing a number of system features supportedby the particular external system.
 15. The non-transitory computerreadable storage medium of claim 11, wherein the set of featurescomprises a feature representing a rate of user interactions of userswith the particular external system.
 16. The non-transitory computerreadable storage medium of claim 10, wherein one or more system defectsinclude unstructured text describing a difference between an implementedfeature of the external system and an expected feature.
 17. Thenon-transitory computer readable storage medium of claim 10, wherein theinstructions further cause the one or more computer processors toperform steps comprising: ranking a plurality of de-duplication rulesbased on the scores of each of the plurality of de-duplication rules;and sending the ranked de-duplication rules for presentation via a userinterface.
 18. The non-transitory computer readable storage medium ofclaim 10, wherein a record describing a system defect comprises one ormore components, wherein each regular expression of a de-duplicationrule is associated with a component of the system defect of thede-duplication rule.
 19. A computer system comprising: one or morecomputer processors; and a non-transitory computer readable storagemedium storing instructions that when executed by the one or morecomputer processors cause the one or more computer processors to performsteps comprising: storing records describing system defects for aplurality of external systems, each system defect associated with anexternal system; receiving, a plurality of de-duplication rules, whereina de-duplication rule is configured to receive a description of an inputsystem defect and determine whether the input system defect is aduplicate of an original system defect associated with thede-duplication rule, the de-duplication rule specifying one or moreregular expressions, wherein the input system defect is determined to bea duplicate responsive to the input system defect matching the one ormore regular expressions; for each of the plurality of de-duplicationrules: determining a set of features comprising, a feature based on anumber of system defects de-duplicated using the de-duplication rules,and determining a score indicating a significance of the de-duplicationrule based on the set of features; identifying a user that submitted thede-duplication rule; and performing an interaction with the userassociated with the de-duplication rule based on the score determinedfor the de-duplication rule.
 20. The computer system of claim 19,wherein the instructions further cause the one or more computerprocessors to perform steps comprising: ranking a plurality ofde-duplication rules based on the scores of each of the plurality ofde-duplication rules; and sending the ranked de-duplication rules forpresentation via a user interface.